[LEADS-444] Support Python 3.14 in lightspeed-evaluation by bsatapat-jpg · Pull Request #262 · lightspeed-core/lightspeed-evaluation

bsatapat-jpg · 2026-06-26T13:01:11Z

Description

Type of change

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

Assisted-by: Claude-4.6-opus-high
Generated by: Cursor

Related Tickets & Documents

Related Issue # https://redhat.atlassian.net/browse/LEADS-444
Closes #

Checklist before requesting a review

I have performed a self-review of my code.
PR has passed all pre-merge test jobs.
If it is a core feature, I have added thorough tests.

Testing

Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

Bug Fixes
- Improved stability in evaluation-related workflows by avoiding a rare startup race condition.
Chores
- Expanded Python support to include 3.14 in CI and packaging metadata.
- Refreshed pinned dependencies across the project, including updates to tracing, NLP, and ML libraries.
- Updated dependency markers and attribution metadata to match the latest resolved environment.

coderabbitai · 2026-06-26T13:01:25Z

Walkthrough

Python 3.14 support is added by updating requires-python to <3.15 and extending the CI test matrix. All four requirement files are regenerated with updated dependency pins and rewritten greenlet environment markers. A new _warmup_instructor_registry() function is added to ragas.py to pre-load the instructor mode registry at import time.

Changes

Python 3.14 support, dependency refresh, and instructor warmup

Layer / File(s)	Summary
Python bounds and CI matrix `pyproject.toml`, `.github/workflows/tests.yaml`	`requires-python` upper bound raised to `<3.15`; Python 3.14 added to the CI test matrix.
instructor registry warmup `src/lightspeed_evaluation/core/llm/ragas.py`	Adds `Mode`, `Provider`, and `mode_registry` imports; introduces `_warmup_instructor_registry()` called at module import time, suppressing `KeyError`.
All-extras export refresh `requirements-all-extras.txt`	Updates pins for `anyio`, `cryptography`, `instructor`, `langchain-`, `opentelemetry-`, `sentry-sdk`, `transformers`, and others; rewrites `greenlet` marker to include `python_full_version < '3.14'`; adds `langfuse==4.12.0` and `wrapt==2.2.2`.
Remaining requirement exports `requirements.txt`, `requirements-local-embeddings.txt`, `requirements-nlp-metrics.txt`	Same set of dependency version bumps and `greenlet` marker rewrites applied across the base and remaining extras requirement files.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding Python 3.14 support to lightspeed-evaluation.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@requirements-local-embeddings.txt`:
- Around line 104-128: The greenlet dependency marker is too restrictive and
blocks Python 3.14 even though greenlet==3.5.3 supports it. Update the
`greenlet==3.5.3` entry in the generated requirements so the
`python_full_version < '3.14'` guard is removed, while keeping any platform
constraints that still apply, and make sure the corresponding generated
requirements files stay consistent.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5170cdd3-bc7e-4e0b-b92d-80ac40749b9f

📥 Commits

Reviewing files that changed from the base of the PR and between 1ae8d2b and 445912e.

⛔ Files ignored due to path filters (2)

uv-gpu.lock is excluded by !**/*.lock
uv.lock is excluded by !**/*.lock

📒 Files selected for processing (7)

.github/workflows/tests.yaml
lsc_agent_eval/pyproject.toml
pyproject.toml
requirements-all-extras.txt
requirements-local-embeddings.txt
requirements-nlp-metrics.txt
requirements.txt

✅ Files skipped from review due to trivial changes (3)

.github/workflows/tests.yaml
pyproject.toml
requirements-all-extras.txt

🚧 Files skipped from review as they are similar to previous changes (1)

lsc_agent_eval/pyproject.toml

asamal4

There are some unrelated changes. I have commented PTAL

asamal4 · 2026-06-29T10:41:36Z

+]
+
+[[package]]
+name = "langfuse"


Why is it being added instead of update ? Did we miss to re-generate lock and requirement files when we added the langfuse as optional dependency ?

Yes, I have missed to push uv.lock file in the langfuse PR's.

Let's add this in a differnt PR

asamal4 · 2026-06-29T10:41:57Z

 logger = logging.getLogger(__name__)


+def _warmup_instructor_registry() -> None:


Why this change ?

While running the evaluation with Python 3.14, I found an issue regarding thread-safety race in instructor's mode registry that crashes. The registry uses dict.pop() on lazy loaders without locking — when our ThreadPoolExecutor threads hit it concurrently, one thread pops the loader while another gets KeyError. Pre-warming it once from the main thread before threads start eliminates the race. Wrapped in try/except for forward compatibility.

Could you please check with previous versions - seems weird that it appeared in 3.14

asamal4

@bsatapat-jpg Please check if all required packages actually support python 3.14 ?

coderabbitai

🧹 Nitpick comments (1)

src/lightspeed_evaluation/core/llm/ragas.py (1)

28-29: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Log the skipped warmup instead of swallowing it silently.

If this workaround stops matching instructor’s behavior, pass leaves no signal in logs. Please emit a debug log here so registry warmup failures remain diagnosable. As per coding guidelines, "Use structured logging with appropriate log levels."

Suggested change

     try:
         mode_registry.get_handlers(Provider.OPENAI, Mode.JSON)
-    except KeyError:
-        pass
+    except KeyError as exc:
+        logger.debug(
+            "Skipping instructor mode registry warmup; handlers are not available yet",
+            exc_info=exc,
+        )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lightspeed_evaluation/core/llm/ragas.py` around lines 28 - 29, The
exception handling in the Ragas warmup path currently swallows a KeyError
silently, which leaves no trace when registry warmup is skipped. Update the
except block in the warmup logic around the instructor-related lookup to emit a
structured debug log with appropriate context instead of using pass. Use the
existing logger in this module (or the same logging pattern used elsewhere in
ragas.py) so the skipped warmup remains diagnosable without changing behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lightspeed_evaluation/core/llm/ragas.py`:
- Around line 28-29: The exception handling in the Ragas warmup path currently
swallows a KeyError silently, which leaves no trace when registry warmup is
skipped. Update the except block in the warmup logic around the
instructor-related lookup to emit a structured debug log with appropriate
context instead of using pass. Use the existing logger in this module (or the
same logging pattern used elsewhere in ragas.py) so the skipped warmup remains
diagnosable without changing behavior.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9e07a319-5b54-4a06-95e2-1742774a3a29

📥 Commits

Reviewing files that changed from the base of the PR and between 445912e and a31c16a.

⛔ Files ignored due to path filters (2)

uv-gpu.lock is excluded by !**/*.lock
uv.lock is excluded by !**/*.lock

📒 Files selected for processing (7)

.github/workflows/tests.yaml
pyproject.toml
requirements-all-extras.txt
requirements-local-embeddings.txt
requirements-nlp-metrics.txt
requirements.txt
src/lightspeed_evaluation/core/llm/ragas.py

✅ Files skipped from review due to trivial changes (2)

pyproject.toml
requirements-nlp-metrics.txt

🚧 Files skipped from review as they are similar to previous changes (4)

.github/workflows/tests.yaml
requirements.txt
requirements-local-embeddings.txt
requirements-all-extras.txt

bsatapat-jpg · 2026-06-29T11:40:01Z

@bsatapat-jpg Please check if all required packages actually support python 3.14 ?

All packages support python 3.14, litellm declares requires-python: <3.14 in its metadata (all versions through 1.90.0), but it's pure Python and works fine on 3.14.

asamal4 · 2026-06-29T12:07:13Z

@bsatapat-jpg Please check if all required packages actually support python 3.14 ?

All packages support python 3.14, litellm declares requires-python: <3.14 in its metadata (all versions through 1.90.0), but it's pure Python and works fine on 3.14.

Two important aspects

works is not same as officially supported
How can this be installed as a package without any workaround

Let's keep this PR open, but we can fix other things in.separate PR

bsatapat-jpg force-pushed the dev branch from 1ae8d2b to 445912e Compare June 29, 2026 05:28

coderabbitai Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread requirements-local-embeddings.txt

bsatapat-jpg force-pushed the dev branch from 445912e to a4d2bca Compare June 29, 2026 06:19

asamal4 reviewed Jun 29, 2026

View reviewed changes

Comment thread requirements-local-embeddings.txt

[LEADS-444] Support Python 3.14 in lightspeed-evaluation

a31c16a

bsatapat-jpg force-pushed the dev branch from a4d2bca to a31c16a Compare June 29, 2026 11:13

coderabbitai Bot reviewed Jun 29, 2026

View reviewed changes

bsatapat-jpg requested a review from asamal4 June 29, 2026 11:43

		logger = logging.getLogger(__name__)


		def _warmup_instructor_registry() -> None:

Uh oh!

Conversation

bsatapat-jpg commented Jun 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Tools used to create PR

Related Tickets & Documents

Checklist before requesting a review

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asamal4 left a comment

Choose a reason for hiding this comment

Uh oh!

asamal4 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

bsatapat-jpg Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

asamal4 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

asamal4 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

bsatapat-jpg Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

asamal4 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asamal4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

bsatapat-jpg commented Jun 29, 2026

Uh oh!

asamal4 commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bsatapat-jpg commented Jun 26, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 26, 2026 •

edited

Loading